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© Object discriminating apparatus and method therefor. 



© An object sensor (15) outputs an obstacle detection signal in response to passing of an obstacle. A central 
control unit (CCU 1) reads a 2-dimensional image from an imaging unit (17) in response to the obstacle 
detection signal. A detection area is obtained by subtracting a background image prestored in a memory (11) 
from the input 2-dimensional image. The detected image on a 2-dimensional' plane is transformed into image 
data in a 3-dimensional space. The CCU (1) computes the projection area of the image date on an arbitrary 
plane in the 3-dimensional space. The object is discriminated by comparing the computed projection area with 
the projection area based on the reference value of the object prestored in the memory (11). 
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Object discriminating apparatus and method therefor 

The present invention relates to an object discriminating apparatus used for discrimination of vehicles, 
men, and the like for, e.g., automatic steering of vehicles, and a method therefor. 

In modern society, vehicles have become indispensable as a transportation means. However, many 
traffic accidents occur by careless driving. For this reason, the study of automatic steering of vehicles has 
s been recently made from the viewpoint of prevention against traffic accidents. 

The study of automatic steering of vehicles is branched into various subjects. One of important subjects 
is concerned with detection and discrimination of an obstacle, wherein an obstacle is detected from a 
photographic image and the type of the obstacle is discriminated. As a conventional method of detecting an 
obstacle using a photographic image, a method of determining an object, which is crossing road boundaries 
w on an image, to be an obstacle is known (32nd (first semiannual conference, 1986) National Conference 
Preliminary Reports 6N-8, pp. 1943 - 1944, Information Processing Society). According to another known 
method, a vehicle traveling ahead is determined by detecting horizontal edges in an image on the basis of 
the fact that the image of a vehicle includes a large number of horizontal edges compared with background 
image (Journal of the Information Processing Society (Japan), vol. 27, No. 7, pp. 663 - 690). 
is In both the above-described methods, however, only an obstacle is detected, but the type of the 
obstacle cannot be discriminated. 

It is an object of the present invention to provide an object discriminating apparatus, which can detect 
an object from a photographic image and discriminate the type of the object, and a method therefor. 

In order to achieve the above object, there is provided an object discriminating apparatus comprising: 
20 image photographing means for photographing an image; object detecting means for detecting an object 
from the photographic image photographed by the image photographing means; area computing means for 
computing a projection area of the object detected by the object detecting means in an arbitrary plane of a 
. 3-dimensional space; and object discriminating means for discriminating the object on the basis of the 
projection area computed by the area computing means. 
25 According to the present invention, a method of discriminating an object comprises the steps of: 
photographing an image; detecting an object from the photographic image, computing a projection area of 
the detected object in an arbitrary plane of a 3-dimensionai space, and discriminating the object on the 
basis of the computed projection area. 

According to the object discriminating apparatus and the method therefor of the present invention, the 
30 projection area of an object in an arbitrary plane of a 3-dimensional space is computed from a photographic 
image. Therefore, even if a distance between a photographing position and an obstacle is changed, an 
actual area can always be calculated, and hence discrimination of an object can be reliably performed. 

This invention can be more fully understood from the following detailed description when taken in 
conjunction with the accompanying drawings, in which: 
35 Fig, 1 is a block diagram showing an object discriminating apparatus according to an embodiment of 

the present invention; 

Fig. 2 is a view illustrating a photographic image obtained by an imaging unit; 
Figs. 3A and 3B are flow charts showing processing of an object detecting section, an area 
computing section, and an object discriminating section in Fig. 1; 
40 Fig. 4 is a view for explaining a method computing the projection area of an object in an arbitrary 

plane of a 3-dimensional space using a photographic image; 

Fig. 5 is a view illustrating coordinate points a', B', C'. and D' in a arbitrary plane of a 3-dimensional 
space when they are respectively set to be wt (ui, vt), W 2 (u 2 , v 2 ),. w 3 , (u 3 , v 3 ), and w* (ui, v*); and 

Fig. 6 is a view illustrating the projection areas of a vehicle and a pedestrian in an arbitrary plane of a 
45 3-dimensional space. 

Fig. 1 is a block diagram showing an object discriminating apparatus according to an embodiment of 
the present invention. Referring to Fig. 1 , central control unit (CCU) 1 comprises object detecting section 3, 
area computing section 5, and object discriminating section 7. Imaging unit 17 is connected to CCU 1. 
Imaging unit 17 forms an image of a target visual field and outputs an image signal. Imaging unit 17 can be 
so constituted by, e.g., an industrial television (ITV) camera such as an automatic focusing type of camera 
which is off the shelf. In addition, read-only memory (ROM) 9 for storing the program of the flow charts 
shown in Figs. 3A and 3B, and random access memory (RAM) 1 1 serving as a work area for performing 
various calculations are connected to CCU 1. CCU 1 can be constituted by. e.g., a microcomputer. CCU 1, 
ROM 9. and RAM 1 1 can be integrated into a one-chip microcomputer. 

Fig. 2 shows an example of image 23 photographed by imaging unit 17. In this embodiment, image 23 
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is constituted by vehicle 19 and pedestrian 21. 

A case wherein vehicle 19 and pedestrian 21 are discriminated in image 23 by using the object 
discriminating apparatus of the present invention will be described below with reference to the flow charts in 
Figs. 3A and 3B. 

In step 31 in Fig. 3A. an image photographed by imaging unit 17 is read. In step 33, it is determined 
whether a detection signal is output from object detecting section 3. If NO is obtained, the flow returns to 
step 31. If YES in step 33. CCU 1 computes number S% of pixels of the image area of the detected object 
in step 35. 

Since vehicle 19 and pedestrian 21 photographed by imaging unit 17 are transformed from 3- 
dimensional space data into 2-dimensional plane date, the 2-dimensional plane data must be perspective^ 
transformed into the 3-dimensional space date. 

If x' • y" is a coordinate system prior to the transformation, and x - y • z is a coordinate system after the 
transformation, transformation from 2-dimensional plane data into 3-dimensional space date can be 
represented by: 



[x y z 11 
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where X" - Y" - Z" - W is a homogeneous coordinate system. 

Equations (1) and (2) are related to equations of transformation when a viewpoint is located at a position 
of z = h on the z-axis in a right-handed coordinate system. The position of Z = h represents a focal 
position of the ITV camera. Since a viewpoint is located on the origin in an eye coordinate system (x e - y 9 - 
z e ), the related equations of transformation in the eye coordinate system can be represented by the 
following equations if x% = y" 9 is a coordinate system prior to the transformation and X* e - Y* e - Z\ - w% is a 
homogeneous coordinate system: 
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.. . (5) 
... (6) 



Therefore, the projection areas of vehicle 19 and pedestrian 21 in an arbitrary plane of a 3-dimensional 
space can be obtained using equations (5) and (6) in the following procedure. Note that plane P is 
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perpendicular to a plane including the optical axis of a camera shown in Fig. 6 and to a plane (ground) with 
whiQh the object is in contact. 

First, in step 37, the coordinate system (x* e - y%) of photographic image 23 is set as shown in Rg. 4, 
and minimum values y" e and x' e , and maximum values y* a and x* e of vehicle 19 and pedestrian 21 are 
5 respectively obtained. In step 39, point A satisfying (at. aH) a (x* e i. y"«i). po*"* b' satisfying (b< +1. b2) 
= (x" 9 2. y"s2). point C satisfying (ci, C2 + 1) = (xVj. v "«3» ancl , P?' nt P satisfying (di-1, d2) = x"^, yV*) are 
obtained. In addition, in step 41, the area (l%) of rectangle ABC D is computed. Area l" B of the rectangle 
having vertexes A , B , C , and D can be given by: 

10 i'e = |(1'2){(x* a2 -x- e1 )(y^ + y- e1 ) 

+ (X' ft3 ' X , e2 )(y- e3 + Vri) 
+ (X'e4 - X' e3 )<y*a4 + y' e3 ) 
+ (X'el ' X*e4)(y\i + V"e4)}| , (7) , 

Subsequently, points a', B , c\ and D on photographic image 23 are transformed and the respective 
rs coordinates of points a', b\ c\ and D' in an arbitrary plane of a 3-dimensional space are obtained. More 
specifically, in step 43, points a', b', c\ and 0 on the image are transformed and corresponding points wi 
(ui, vi), w 2 (U2, v 2 ), w 3 (u 3 , v 3 ), and w 4 (u*. v 4 ) in the arbitrary plane shown in Fig. 5 are computed. In 
addition. CCU 1 calculates the area (I) of rectangle wi , W2, w 3 , wt in the arbitrary plane. Area I of rectangle 
wi, W2, w 3f Wi can be calculated by the following equation: 

20 

I = |(1/2){(U 2 -Ui)(V 2 +■ Vi) 

' + (u 3 - u 2 )(v 3 + v 2 ) 

+ (Ut ' U 3 )(Vi + v 3 ) 

+ (ut - u*)(vi + vi)}| (8) 

25 

In step 47, CCU 1 calculates the projection area of the object on an arbitrary plane by the following 
equations: 

S = S\ •(!/!%) (9) 

3Q 

In step 49, CCU 1 discriminates the object on the basis of the projection area on the arbitrary plane. 
This processing is performed, for example, such that a standard projection area for each type of vehicle and 
a standard projection area of pedestrians are stored in RAM 1 1 and CCU 1 discriminates an object by 
comparing a calculated projection area with a projection area read out from RAM 11. Discrimination of an 

35 object may be performed by determining whether a projection area on an arbitrary plane of a 3-dimensional 
space exceeds a given threshold value or not. Upon comparison with a threshold value, even when only a 
vehicle or a pedestrian is photographed in a photographic image, the type of an obstacle can be 
discriminated. In the above-described embodiment, an object is detected by storing the density value of a 
road in advance and using it as a threshold value. However, an object may be detected by other known 

4a techniques, e.g., detecting horizontal edges in an image. 



Claims 

45 1. An object discriminating apparatus, comprising: 
image capturing means (17) for capturing an image; 

object detecting means (3) for detecting an object from the image captured by said capturing means; 
area computing means (5) for computing a projected area of the object detected by said object detecting 
means on an arbitrary plane of a 3-dimensionai space where the object is located; and 
so object discriminating means (7) for discriminating the object on the basis of the projected area computed by 
said area computing means. 

2. An apparatus according to claim 1, characterized in that said area computing means (5) transforms 2- 
dimensional plane data in an image area of the object detected by said object detecting means into 3- 
dimensional space data according to the following equations: 



4 



BNSDOCID: <EP. 



.03(W042A2_L> 



EP 0 304 042 A2 



to 



rs 



20 



25 



30 



35 



40 



45 



SO 



55 



[x e y e z e 1] 
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0 10 0 
0 0 2 1/h 
L 0 0 -h 0 



e 1 e 6 e 
x* e - (X* e /w* e ) 
= h»(x e /z e ) 

Y*e = U*e/"*e) 
= h»(y e /z e ) 



...d) 
... (2) 
...(3) 



(where x e - y 9 - Z e is an eye coordinate system. x' a • y. is a coordinate system prior to transformation, and 
X* - Y* - T - W is a homogeneous coordinate system), obtains minimum and maximum y*. values, and 
minimum and maximum x". values in the extracted image area, obtains area 1% of a rectangle in an .mage 
plane having points A, B*. C', and o' as vertexes which are obtained by the following equation: 

I'e =|(1/2){(x' e 2-X- 9l )(ye 2 + V%i) 

+ (X' e3 * X- e2 )(y e 3 + y'ez) 

+ (X'e4 • X" e3 )(y e - + y"e3) 

+ (X' e , - S'e4)(y' e , + ye4>}| (4) 

(where point A having the minimum y. value is set as A = (a,, a 2 ), point B having the maximum x'. value 
is set as B Mb,. b 2 ), point C having the maximum y' a value is set as C = (c, c 2 ), and point D having the 
minimum x% value is set as D = <d,. d 2 ), and a point (a,, a 2 -D is set as A = x\„ y„). a po.nt (b, +1. b 2 ) 
issetasB' = x% 2 , y*). a point (c. c 2 + 1) is set as C = (x% 3 , Ves). a point (dH, d 2 ) .s set as i0 i - (x^ 
v'.)) obtains area I of a rectangle having points w,, w 2 . w 3 , and w*. as vertexes us.ng the following 
equation when each point of A . B C', and D projected on an arbitrary plane of a 3-dimensional space are 
respectively set to be w,(ui, v,), w 2 (u 2 . v 2 ), w 3 (u 3 , v 3 ), and w t (u. v*): 

I = | (1/2){(U2 ■ u,)(v 2 + v,) 
+ (u 3 - u 2 )(v 3 + v 2 ) 

+ (Ut - U 3 )(Vi + v 3 ) 

+ (ui - ut)(v, + v t )}| (5) 

and calculates a projected area of the detected object on the arbitrary plane of the 3-dimensional space 
using the following equation: 

S = SV(l/l'e) (6) 

where S% is a number of pixels of an extracted image. ..... 

3 An apparatus according to claim 1, characterized in that said object discriminating means (7) 
comprises reference area value storing means for storing a reference value of an object to be discriminated, 
and discriminates the object by comparing the area obtained by said area computing means with the 
reference value read out from said reference area value storing means. 

4. A method of discriminating an object, comprising the steps of: 
capturing an image: 

detectinq an object from the captured, image; 

computing a projected area of the detected object on an arbitrary plane of a 3-dimens.onal space where the 

object is located; and 

discriminatina the object on the basis of the computed projection area. 

rrmethod according to claim 4, characterized in that the step of detecting the object compn.es the 

^restoring a background image, which is an image before the object is captured, in a memory; and 
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detecting existence of the object using a sensor, capturing an image including the object in response to an 
object detection signal from said sensor, and detecting the object by subtracting the background image 
prestored in said memory from the captured image. 

6. A method according to claim 4, characterized in that the step of computing the area comprises the 
5 steps of transforming 2-dimensional plane data in an image area of the object detected by said object 
detecting means into 3-dimensional space data according to the following equations: 
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(where x e - y 9 - z e is an eye coordinate system, x" a - y* e is a coordinate system prior to transformation, and 
X" e - Y% * Z% * w *e is a homogeneous coordinate system), obtaining minimum and maximum y*, values and 
minimum and maximum x" e values in the extracted image area, obtaining area 1% of a rectangle in 2- 
dimensional image data having points A\ B' ( C, and as vertexes which are obtained by the following 
equation: 

I*. -Id^Wx^-xVKy'rt + y" el ) 

+ (x* 93 - x- a2 )(y e3 + y' e2 ) 

+ (x"e4 - x' e3 )(yVi + y a3 ) 

+ (x'.i - x^Xy'e, + yVt)} I (4) 

(where point A having the minimum y" 9 value is set as A = (at, a 2 ), point B having the maximum x% value 
is set as B = (bi, b2>, point C having the maximum y' e value is set as C = (ct , C2), and point D having the 
minimum x* e value is set as D = (d?. d2), and a point (ai, a2-1) is set as a' = (x # et , y* a1 ), a point (bi +1, b 2 ) 
is set as B = (x' e2 , y B e2 ), a point (Ci, c 2 + 1) is set as c' = (x* e3 , y' e3 ) t and a point (di-1, da) is set as O' = 
(x* M . y"e4)). obtaining area I of a rectangle having points w 1( w 2l w 3l and w 4 as vertexes using the following 
equation when each point of a\ b' ( C', and D' projected on an arbitrary plane of a 3-dimensional space are 
respectively set to be wi (ui, v t ), w 2 (U2, v 2 ), w 3 (u 3 . v 3 ). and w* (ua, vi): 

I »|(1/2){(U2-Ui)(V 2 + V-) 

+ (u 3 - u 2 )(v 3 + v 2 ) 

+ (Ua - U 3 )(U* + V3) 

+ (Ut - u*)(vi + vt)}| (5) 



and calculating a projected area of the detected object on the arbitrary plane of the 3-dimensional space 
using the following equation: 
S = SV(l/re) (6) 

where S% is a number of pixels of an extracted image. 

7. A method according to claim 4, characterized in that the step of discriminating the object comprises 
the steps of prestoring a reference value of the object to be discriminated in a reference area value 
memory, and comparing the area obtained by the step of computing the area with the reference value read 
out from said reference area value memory. 
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© An object sensor (15) outputs an obstacle detec- 
tion signal in response to passing of an obstacle. A 
centra! control unit (CCU 1) reads a 2-dimensional 
image from an imaging unit (17) in response to the 
obstacle detection signal. A detection area is ob- 
tained by subtracting a background image prestored 
in a memory (11) from the input 2-dimensional im- 
age. The detected image on a 2-dimensional plane 
is transformed into image data in a 3-dimensional 
space. The CCU (1) computes the projection area of 
the image date on an arbitrary plane in the 3- 
dimensional space.' The object is discriminated by 
comparing the computed projection area with the 
projection area based on the reference value of the 
object prestored in the memory (11). 
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